IsoBN: Fine-Tuning BERT with Isotropic Batch Normalization

Fine-tuning pre-trained language models (PTLMs), such as BERT and its better variant RoBERTa, has been a common practice for advancing performance in natural language understanding (NLU) tasks. Recent advance in representation learning shows that isotropic (i.e., unit-variance and uncorrelated) embeddings can significantly improve performance on downstream tasks with faster convergence and better generalization. The isotropy of the pre-trained embeddings in PTLMs, however, is relatively under-explored. In this paper, we analyze the isotropy of the pre-trained [CLS] embeddings of PTLMs with straightforward visualization, and point out two major issues: high variance in their standard deviation, and high correlation between different dimensions. We also propose a new network regularization method, isotropic batch normalization (IsoBN) to address the issues, towards learning more isotropic representations in fine-tuning by dynamically penalizing dominating principal components. This simple yet effective fine-tuning method yields about 1.0 absolute increment on the average of seven NLU tasks.Comment: AAAI 202

SNR Enhancement in Brillouin Microspectroscopy using Spectrum Reconstruction

Brillouin imaging suffers from intrinsically low signal-to-noise ratios (SNR). Such low SNRs can render common data analysis protocols unreliable, especially for SNRs below $\sim10$. In this work we exploit two denoising algorithms, namely maximum entropy reconstruction (MER) and wavelet analysis (WA), to improve the accuracy and precision in determination of Brillouin shifts and linewidth. Algorithm performance is quantified using Monte-Carlo simulations and benchmarked against the Cram\'er-Rao lower bound. Superior estimation results are demonstrated even at low SNRS ($\geq 1$). Denoising was furthermore applied to experimental Brillouin spectra of distilled water at room temperature, allowing the speed of sound in water to be extracted. Experimental and theoretical values were found to be consistent to within $\pm1\%$ at unity SNR

Research on Experiential Marketing Strategy Based on the Sale of Baking Products

To understand the consumer is the prerequisite for the enterprise to enter the consumer market. With the market competition intensified, Traditional marketing strategy is difficult to achieve the expected goal of the enterprise. It needs new marketing theory to guide the market and satisfy the demand. In this paper, the sales of baking products are taken for an example in which 425 questionnaires analyze five models of emotion, culture, service, environment and personality in detail and the paper suppllies some corresponding suggestions & measure to tackle the problems as the lack of large-scale leading enterprises and Enterprise homogeneity

Multivariate analysis of Brillouin imaging data by supervised and unsupervised learning

Brillouin imaging relies on the reliable extraction of subtle spectral information from hyperspectral datasets. To date, the mainstream practice has been using line fitting of spectral features to retrieve the average peak shift and linewidth parameters. Good results, however, depend heavily on sufficient SNR and may not be applicable in complex samples that consist of spectral mixtures. In this work, we thus propose the use of various multivariate algorithms that can be used to perform supervised or unsupervised analysis of the hyperspectral data, with which we explore advanced image analysis applications, namely unmixing, classification and segmentation in a phantom and live cells. The resulting images are shown to provide more contrast and detail, and obtained on a timescale $10^2$ faster than fitting. The estimated spectral parameters are consistent with those calculated from pure fitting

Unsupervised Cross-Task Generalization via Retrieval Augmentation

Humans can perform unseen tasks by recalling relevant skills that are acquired previously and then generalizing them to the target tasks, even if there is no supervision at all. In this paper, we aim to improve such cross-task generalization ability of massive multi-task language models such as T0 (Sanh et al., 2021) in an unsupervised setting. We propose a retrieval-augmentation method named ReCross that takes a few unlabelled examples as queries to retrieve a small subset of upstream data and uses them to update the multi-task model for better generalization. Our empirical results show that the proposed ReCross consistently outperforms non-retrieval baselines by a significant margin.Comment: Project website: https://inklab.usc.edu/ReCross